CM-GANs: Cross-modal Generative Adversarial Networks for Common Representation Learning

It is known that the inconsistent distribution and representation of different modalities, such as image and text, cause the heterogeneity gap, which makes it very challenging to correlate such heterogeneous data. Recently, generative adversarial networks (GANs) have been proposed and shown its strong ability of modeling data distribution and learning discriminative representation, and most of the existing GANs-based works mainly focus on the unidirectional generative problem to generate new data such as image synthesis. While we have completely different goal, which aims to effectively correlate existing largescale heterogeneous data of different modalities, by utilizing the power of GANs to model the cross-modal joint distribution. Thus, in this paper we propose Cross-modal Generative Adversarial Networks (CM-GANs) to learn discriminative common representation for bridging the heterogeneity gap. The main contributions can be summarized as follows: (1) Cross-modal GANs architecture is proposed to model the joint distribution over the data of different modalities. The inter-modality and intramodality correlation can be explored simultaneously in generative and discriminative models. Both of them beat each other to promote cross-modal correlation learning. (2) Cross-modal convolutional autoencoders with weight-sharing constraint are proposed to form the generative model. They can not only exploit the cross-modal correlation for learning the common representation, but also preserve the reconstruction information for capturing the semantic consistency within each modality. (3) Cross-modal adversarial mechanism is proposed, which utilizes two kinds of discriminative models to simultaneously conduct intra-modality and inter-modality discrimination. They can mutually boost to make the generated common representation more discriminative by adversarial training process. To the best of our knowledge, our proposed CM-GANs approach is the first to utilize GANs to perform cross-modal common representation learning, by which the heterogeneous data can be effectively correlated. Extensive experiments are conducted to verify the performance of our proposed approach on cross-modal retrieval paradigm, compared with 10 state-of-the-art methods on 3 cross-modal datasets.